internal/gps: ensure packages are deducible before attempting to solve #697

ibrasho · 2017-06-01T01:20:36Z

An attempt to resolve #581.

I still think it could use some polishing, but I wanted to check if I'm moving in the right direction.

sdboyer

Yep! Definitely the right direction.

One thing, though. I realize I'm countermanding myself, but let's actually do this under solver.Solve() instead of Prepare(). These deductions have the possibility of triggering network activity, and I'd prefer to keep the work done in Prepare() strictly IO-free.

It'd be fine to add a solver.validateParams() method that does this, and call it nearly first thing, immediately before solver.selectRoot() in solver.Solve(). Chances are we'll have some more validation like this to add, and we can lump it into that method.

ibrasho · 2017-06-01T11:19:34Z

@sdboyer Done. Note that Prepare() doc says the following:

// This function reads and validates the provided SolveParameters. If a problem
// with the inputs is detected, an error is returned. Otherwise, a Solver is
// returned, ready to hash and check inputs or perform a solving run.

I've extracted the logic to a solver. validateParams() method. But I think it belongs to Prepare(), or that the comment should be clarified.

ibrasho · 2017-06-01T20:48:34Z

@sdboyer It seems that bridge.DeduceProjectRoot() allows race conditions.
solver.mtr is not protected against simultaneous writes by multiple goroutines, and it's not possible to get the gps.SourceManager directly to avoid using bridge.DeduceProjectRoot().

I tried to go back to Prepare(), but noticed that passing arbitrary packages to Prepare in tests fails.

Any preference on how to tackle this?
Adding a mutex over Solver.mtr in bridge.DeduceProjectRoot() will render goroutines useless since it will force sequential execution. I'm thinking of exposing the gps.SourceManager to the gps.solver.

sdboyer · 2017-06-02T00:31:00Z

@ibrasho ahh...crap.

Yes, the bridge allows races, because the entire solver is designed to be strictly single-threaded.

OK, given these issues, yes - let's move the verification back to Prepare(). I'm confused by that link you provide, though - I don't see any failures there?

ibrasho · 2017-06-02T00:50:38Z

@sdboyer That test case adds 2 dependencies called baz and quz. And since they don't exist, Prepare() will return an error and the test will fail. In TestHashInputsOverrides, c is used multiple times too, causing failures.

This is what's causing CI builds to fail.

sdboyer · 2017-06-02T02:53:01Z

🤦‍♂️

sorry, of course, that make sense.

let me ponder a bit on the least invasive way to fix this.

sdboyer · 2017-06-02T18:56:37Z

OK, new plan. We don't actually need to do this validation within the solver, because it's all failures we'd arrive at eventually, anyway. That's a strong indication that this isn't actually "validation" in the sense that these conditions must be met for solver behavior to be defined. It's actually just preemptively doing work the solver would do later.

Let's do this validation as a standalone function in gps, and call that function directly from dep, prior to Solve()ing. That'll also give us more reusability, which we'll probably need to employ later within sourceGateway.getManifestAndLock(). We need to guard against this there, too - but in a separate PR.

Returning to the original panic encountered in #581, we can do yet another separate PR which is about gps' purely internal graceful handling of that case.

ibrasho · 2017-06-05T23:33:00Z

@sdboyer: This is passing now. But I still don't like it.

I'm creating another instance of gps.rootdata since it's not exported in the solver created by gps.Prepare().

sdboyer

sorry, i know i've been slow the last week - trying to catch up!

sdboyer · 2017-06-14T15:30:45Z

internal/gps/solver.go

+}
+
+// ValidateParams validates the solver parameters to ensure solving can be completed.
+func ValidateParams(params SolveParameters, sm SourceManager) error {


Need just a basic test for this - feed it some SolveParameters with both valid and invalid import paths.

Are the tests in solver_test.go enough?

sorry, yes, they are. but, how about we rename that file to solver_inputs_test.go, and move TestBadSolveOpts into there?

Add gps.ValidateParams to ensure all packages in SolverParams are deducible. Signed-off-by: Ibrahim AshShohail <ibra.sho@gmail.com>

Signed-off-by: Ibrahim AshShohail <ibra.sho@gmail.com>

sdboyer · 2017-07-26T10:12:42Z

i think the CLA issue should clear if a new commit comes in. maybe just rebase this and re-push, that should be enough.

ibrasho · 2017-07-27T17:26:48Z

That trick actually works. 😁

sdboyer

little nit, looks like a debugging fmt.Println() survived

sdboyer · 2017-07-28T05:15:10Z

internal/gps/solver.go

+	}
+
+	deducePkg := func(ip string, sm SourceManager) {
+		fmt.Println(ip)


seems like a debugging print?

Signed-off-by: Ibrahim AshShohail <ibra.sho@gmail.com>

ibrasho · 2017-08-04T20:08:21Z

Anything else needed here? 😁

sdboyer

nothing! in we go 😄

sdboyer · 2017-08-05T04:03:34Z

see, i said that, but then...yeah, we need this to be expanded, now that there are more places we actually solve/prepare params in ensure.

googlebot added the cla: yes label Jun 1, 2017

ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch 2 times, most recently from bbde9bb to da33cb1 Compare June 1, 2017 01:25

sdboyer reviewed Jun 1, 2017

View reviewed changes

ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch 2 times, most recently from c7a2389 to e159914 Compare June 1, 2017 19:55

sdboyer mentioned this pull request Jun 2, 2017

dep ensure fails on subpackage with a revision constraint #705

Closed

ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch 2 times, most recently from b89b600 to 2058950 Compare June 3, 2017 17:25

ibrasho mentioned this pull request Jun 13, 2017

"unable to read metadata" error #749

Closed

sdboyer suggested changes Jun 14, 2017

View reviewed changes

ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch from fc84b7b to f1d5b48 Compare June 15, 2017 20:29

internal/gps: ensure packages are deducible before attempting to solve

a961d76

Add gps.ValidateParams to ensure all packages in SolverParams are deducible. Signed-off-by: Ibrahim AshShohail <ibra.sho@gmail.com>

ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch from f1d5b48 to a961d76 Compare July 16, 2017 22:28

ibrasho mentioned this pull request Jul 16, 2017

Ensure we're correct on case insensitivity (for github...?) #433

Closed

sdboyer mentioned this pull request Jul 18, 2017

dep double clones goconvey #843

Closed

sdboyer closed this Jul 19, 2017

sdboyer reopened this Jul 19, 2017

ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch 2 times, most recently from e8d34d0 to a961d76 Compare July 21, 2017 00:40

rename internal/gps/solver_test.go -> internal/gps/solver_inputs_test.go

4a2215b

Signed-off-by: Ibrahim AshShohail <ibra.sho@gmail.com>

sdboyer mentioned this pull request Jul 21, 2017

gps panics if it is unable to determine repository type #817

Closed

sdboyer closed this Jul 26, 2017

sdboyer reopened this Jul 26, 2017

ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch from 1f42a17 to b6260d6 Compare July 27, 2017 17:26

sdboyer suggested changes Jul 28, 2017

View reviewed changes

internal/gps/solver.go Outdated

}

deducePkg := func(ip string, sm SourceManager) {

fmt.Println(ip)

Copy link

Member

sdboyer Jul 28, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like a debugging print?

gps: move TestBadSolveOpts to solver_inputs_test.go

92df3dc

Signed-off-by: Ibrahim AshShohail <ibra.sho@gmail.com>

ibrasho force-pushed the ensure-packages-are-deducible-before-solving branch from b6260d6 to 92df3dc Compare July 29, 2017 06:41

sdboyer approved these changes Aug 5, 2017

View reviewed changes

sdboyer merged commit 2e56bed into golang:master Aug 5, 2017

ibrasho mentioned this pull request Aug 9, 2017

cmd/dep: always ensure that gps.SolveParameters are validated #981

Merged

sdboyer mentioned this pull request Aug 9, 2017

dep ensure pulls malformed subpackage #984

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

internal/gps: ensure packages are deducible before attempting to solve #697

internal/gps: ensure packages are deducible before attempting to solve #697

ibrasho commented Jun 1, 2017 •

edited

sdboyer left a comment

ibrasho commented Jun 1, 2017 •

edited

ibrasho commented Jun 1, 2017 •

edited

sdboyer commented Jun 2, 2017

ibrasho commented Jun 2, 2017 •

edited

sdboyer commented Jun 2, 2017

sdboyer commented Jun 2, 2017

ibrasho commented Jun 5, 2017

sdboyer left a comment

sdboyer Jun 14, 2017

ibrasho Jul 16, 2017

sdboyer Jul 19, 2017

ibrasho Jul 21, 2017

sdboyer commented Jul 26, 2017

ibrasho commented Jul 27, 2017

sdboyer left a comment

sdboyer Jul 28, 2017

ibrasho commented Aug 4, 2017

sdboyer left a comment

sdboyer commented Aug 5, 2017

internal/gps: ensure packages are deducible before attempting to solve #697

internal/gps: ensure packages are deducible before attempting to solve #697

Conversation

ibrasho commented Jun 1, 2017 • edited

sdboyer left a comment

Choose a reason for hiding this comment

ibrasho commented Jun 1, 2017 • edited

ibrasho commented Jun 1, 2017 • edited

sdboyer commented Jun 2, 2017

ibrasho commented Jun 2, 2017 • edited

sdboyer commented Jun 2, 2017

sdboyer commented Jun 2, 2017

ibrasho commented Jun 5, 2017

sdboyer left a comment

Choose a reason for hiding this comment

sdboyer Jun 14, 2017

Choose a reason for hiding this comment

ibrasho Jul 16, 2017

Choose a reason for hiding this comment

sdboyer Jul 19, 2017

Choose a reason for hiding this comment

ibrasho Jul 21, 2017

Choose a reason for hiding this comment

sdboyer commented Jul 26, 2017

ibrasho commented Jul 27, 2017

sdboyer left a comment

Choose a reason for hiding this comment

sdboyer Jul 28, 2017

Choose a reason for hiding this comment

ibrasho commented Aug 4, 2017

sdboyer left a comment

Choose a reason for hiding this comment

sdboyer commented Aug 5, 2017

ibrasho commented Jun 1, 2017 •

edited

ibrasho commented Jun 1, 2017 •

edited

ibrasho commented Jun 1, 2017 •

edited

ibrasho commented Jun 2, 2017 •

edited